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METHOD AND APPARATUS FOR PERFORMING TONE SCALE 
MODIFICATIONS ON A SPARSELY SAMPLED EXTENDED DYNAMIC 
RANGE DIGITAL IMAGE 

FIELD OF THE INVENTION 

The invention relates generally to the field of image capture, and 
more specifically to a method of performing tone scale modifications on an 
extended dynamic range digital image. 

BACKGROUND OF THE INVENTION 

Image sensing devices, such as a charge-coupled device (CCD), are 
commonly found in such products as digital cameras, scanners, and video 
cameras. These image sensing devices have a limited dynamic range when 
compared to traditional photographic film products. A typical electronic image 
sensing device has a dynamic range of about 7 stops. This means that the 
exposure for a typical scene must be determined with a fair amount of accuracy in 
order to avoid clipping the resultant signal. By contrast, natural scenes often 
exhibit a dynamic range of 9 stops and higher. This is mainly a consequence of 
multiple light sources with widely varying intensities illuminating the scene 
objects. Specular highlights also contribute to the dynamic range of natural 
scenes. 

Electronic sensors used to scan photographic film must also 
contend with a high dynamic range of signal intensities. In U.S. Patent No. 
5,221,848 issued June 22, 1993 to Milch entitled High Dynamic Range Film 
Digitizer and Method of Operating the Same discloses a method and apparatus 
designed to extend the dynamic range of an electronic image sensor. Intended 
primarily for scanning photographic film, the system described by Milch includes 
a one-pass film scanner using a charge-coupled device scanner having a plurality 
of linear arrays having the same spectral sensitivity. One of the arrays has a faster 
response to light than the other array. The information from the two arrays is then 
combined and digitized forming an extended dynamic range digital image. 
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Digital electronic cameras employ a single image sensor with a 
color filter array (CFA) to produce a sparsely sampled digital image. A typical 
color filter array pattern is disclosed in U.S. Patent No. 3,971,065 issued July 20, 
1976 to Bayer entitled Color Imaging Array. Interpolation algorithms are 
5 employed to produce a full resolution color image from the sparsely sampled 
image. Digital cameras also need to record scenes having a high dynamic range. 
One way to obtain a high dynamic range image from a digital camera is to employ 
a high bit depth analog to digital converter in the camera. Another way is to 
employ an image sensor having interspersed fast and slow photosites as disclosed 

10 in copending U.S. Serial No. 09/615,398 filed July 13, 2000 by Gallagher et al. 
which is incorporated herein by reference. 

It is known to employ a tone scale function that is specifically 
designed for use with extended dynamic range digital images. For example, US 
Patent No. 5,822,453 issued October 13,1998 to Lee et al. entitled Method for 

1 5 Estimating and Adjusting Digital Image Contrast discloses a method of 

calculating a tone scale function using the pixel values of an extended dynamic 
range digital image involving estimating the scene contrast from the digital image. 
However, the method taught by Lee et al. is not optimized for images that are 
captured by extended dynamic range image sensors having fast and slow 

20 photosites. 

Therefore, there exists a need for an improved method of 
performing tone scale modifications on extended dynamic range images of the 
type that are captured by extended dynamic range image sensors having fast and 
slow photosites. 

25 

SUMMARY OF THE INVENTION 

The need is met according to the present invention by providing a 
method of generating a tone scale function for a sparsely sampled extended 
dynamic range digital image that includes the steps of: providing a sparsely 
30 sampled extended dynamic range image sensing device having fast photosites with 
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a predetermined response to light exposure interspersed with slow photosites with 
a slower response to the same light exposure; using the image sensing device to 
produce a sparsely sampled high resolution digital image having fast pixel values 
produced by the fast photosites and slow pixel values produced by the slow 
5 photosites; and generating the tone scale function using only slow pixel values 
from the sparsely sampled high resolution digital image. 

According to one aspect of the invention, the tone scale function is 
applied to the sparsely sampled extended dynamic range digital image prior to 
interpolating the digital image to a full resolution digital image. 

10 

ADVANTAGES 

The present invention has the advantage that the tone scale function 
is generated with a subset of pixel values thereby reducing the computation time 
for generating the tone scale function. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of a digital imaging system utilizing an 
extended-range image sensing device and a digital image processor according to 
the invention; 

20 Fig. 2 is a functional block diagram of the digital image processor; 

Fig. 3 is a cross-sectional view of an interline image sensing device 
employing an array of lenslets to alter the response of selected photosites; 

Fig. 4 is a cross-sectional view of a full frame image sensing device 
employing a metal mask to alter the response of selected photosites; 
25 Fig. 5 is a graph illustrating the response of a fast photosite and a 

slow photosite; 

Fig. 6 is a cross-sectional view of an image sensing device 
employing an array of neutral density filters to alter the response of selected 
photosites; 
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Fig. 7 A illustrates the arrangement of slow photosites and fast 
photosites on a panchromatic image sensing device; 

Fig. 7B illustrates the arrangement of slow photosites and fast 
photosites on a color image sensing device; 
5 Fig. 8 Shows a block diagram of the dynamic range extending filter 

array (DREFA) processor; 

Fig. 9 Shows a block diagram of the paxelization module; 

Fig. 10 shows an example of the arrangement of pixels for block 

averaging; and 

10 Fig. 1 1 is a functional block diagram the enhancement processor. 

DETAILED DESCRIPTION OF THE INVENTION 

A digital image is comprised of a one or more digital image 
channels. Each digital image channel is comprised of a two-dimensional array of 

1 5 pixels. Each pixel value relates to the amount of light received by an imaging 
capture device corresponding to the geometrical domain of the pixel. For color 
imaging applications a digital image will typically consist of red, green, and blue 
digital image channels. Other configurations are also practiced, e.g. cyan, 
magenta, and yellow digital image channels. For monochrome applications, the 

20 digital image consists of one digital image channel. Motion imaging applications 
can be thought of as a time sequence of digital images. Those skilled in the art 
will recognize that the present invention can be applied to, but is not limited to, a 
digital image for any of the above mentioned applications. 

Although the present invention describes a digital image channel as 

25 a two dimensional array of pixel values arranged by rows and columns, those 

skilled in the art will recognize that the present invention can be applied to mosaic 
(non rectilinear) arrays with equal effect. Those skilled in the art will also 
recognize that although the present invention describes replacing an original pixel 
value with processed pixel values, it is also contemplated to form a new digital 

30 image with the processed pixel values and retain the original pixel values. 
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Imaging devices employing electronic sensing devices are well 
known, therefore the present description will be directed in particular to elements 
forming part of, or cooperating more directly with, apparatus in accordance with 
the present invention. Elements not specifically shown or described herein may 
5 be selected from those known in the art. Note that as used herein, the term image 
is a two dimensional array of values. An image may be a two dimensional subset 
of another image. The present invention is preferably implemented using a 
programmed digital computer. The computer can be a general purpose computer, 
such as a personal computer, or a special purpose computer designed for image 
10 processing. It is within the ordinary skill of the programming art to produce a 
computer program for practicing the present invention based on the following 
disclosure. 

The essential elements employed in the practice of the present 
invention are shown as a functional block diagram in Fig. 1. Light from an object 

15 or scene is incident upon a lens 2, forming a photographic image on an extended 
dynamic range image sensing device 10 such as a charged-coupled device (CCD) 
with a color filter array (CFA) having fast and slow photosites as described below. 
Note that other devices, such as CMOS devices, may be used as the image sensing 
device 10. The image sensing device 10 is a sparsely sampled, extended dynamic 

20 range image sensing device as will be described in further detail below. An 

optical low pass filter 6 placed between the lens 2 and the image sensing device 
10, performs a slight blurring of the imaged light in order to reduce the occurrence 
of aliasing. An AID converter 14 receives the voltage signal corresponding to the 
imaged light from the image sensing device 10 and produces an image signal 

25 corresponding to the voltage signal. The digital image processor 200 receives the 
image signal from the A/D converter 14, modifies the image signal and produces a 
digital image. As noted above, the digital image processor 200 can be a 
programmed personal computer, or a special purpose image processor. The user 
of the digital imaging system can make selections, with a user selection device 62 

30 such as a keyboard, mouse, or button on a camera body, with regard to the tonal 
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appearance of an image. For example, the user may view an image created with a 
certain tone scale function on a display device 64. The user may then specify a 
desired tone scale adjustment. For example, this desired tone scale adjustment 
may be specified by the user indicating with the user selection device 62 that the 
5 shadows should be lightened. An adjusted tone scale function is then created 
using the desired tone scale adjustment. Creating and modifying a tone scale 
function based on desired tone scale adjustments from a user is well known in the 
art of image processing. For example, commonly assigned U.S. Patent No. 
5,012,333 by Lee et al. describes an interactive method for generating a tone scale 

10 function and applying the function to an image. The present invention can also be 
implemented within a digital camera. For this digital imaging system application, 
an enhanced digital image derived from a low resolution digital image is displayed 
on a display device 64, for example a liquid crystal display device (LCD) as a 
component of the digital camera. Alternatively, the present invention can be 

1 5 practiced in a film or reflection scanner or other device that produces an extended 
dynamic range digital image. 

The A/D converter 14 shown in Fig. 1 converts the voltage signal 
produced by the image sensing device 10 into an image signal, i.e. a stream of 
digital pixel values corresponding to the voltage signal produced by the photosites 

20 of the image sensing device 10. More specifically, the A/D converter 14 converts 
the voltage signal, nearly linear with respect to the intensity of the incident light, 
from the image sensing device 10 to a discrete digital image signal, e.g. a 10 bit 
signal where the linear encoded values range from 0 to 1023. The A/D converter 
14 may also perform processing to convert the linear code value domain image 

25 signal to a nonlinear code value domain image signal, such as an 8 bit logarithmic 
signal as is commonly performed in the art. For example, the following equation 
can be used to convert a 10 bit linear image signal a(x,y), where (x,y) specifies the 
row and column index of the signal location with reference to the image sensing 
device 10, into an 8 bit logarithmic image signal b(x,y): 
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f 0 0<a(x,y)<31 

b(x,y) = ^ (1) 
[73.5975 In a(x,y)- 255 32 < a(x,y) < 1024 v 

Note that each stop of exposure (in the linear response region of the image sensing 

device) results in a doubling of the linear image signal a(x,y) and results in an 

increase of the logarithmically encoded image signal b(x,y) by 51. In this case, the 

5 value 5 1 represents the number of code values per stop (cvs) of exposure. 

The digital image processor 200 shown in Fig. 1 is illustrated in 

more detail in Fig. 2. The image signal is received by the dynamic range 

extending filter array (DREFA) processor 22. The DREFA processor 22 

processes the sparsely sampled high resolution digital image output from the A/D 

1 0 converter 14 by expanding the dynamic range of the image and interpolating the 
sample values. The DREFA processor 22 then transmits the modified image 
signal to the CFA interpolator 26 where the color values are interpolated to 
provide a color value at each pixel. The output of the CFA interpolator 26 is a full 
resolution digital image. The paxelization module 220 also receives the sparsely 

1 5 sampled high resolution digital image and produces a paxelized digital image, i.e. 
a low resolution digital image derived from the image signal. Optionally, the high 
resolution digital image is also received by an enhancement processor 240 which 
receives the paxelized digital image, calculates enhancements based on the 
paxelized digital image and applies the enhancements to the pixels of the high 

20 resolution digital image to generate an enhanced digital image. 

The purpose of the CFA interpolator 26 is to generate a full 
description of the color for each location of the sensed photographic image. In the 
preferred embodiment, the image sensing device 10 consists of an array of 
photosensitive elements called photo sites. Each photosite is typically provided 

25 with either a red, green, or blue filter, as described by Bayer in commonly 

assigned U.S. Patent No. 3,971,065, which is incorporated herein by reference. 
The Bayer array is a color filter array in which green filters are located in a 
checkerboard pattern over the photosites with red and blue filters alternating line 
by line to fill the interstices of the checkerboard pattern; this produces twice as 



many green filter sites as either red or blue filter sites. Note that the method 
described herein may be easily extended to color filter arrays with different 
arrangements of the primaries, a different number of primaries, or a different set of 
primaries. Thus, in the preferred embodiment, each photosite is sensitive to either 
5 red, green, or blue light. However, it is desirable to obtain a pixel value 

corresponding to an exposure for each of the red, green, and blue exposures at 
each photosite location. The pixel values of the sparsely sampled high resolution 
digital image output from the A/D converter 14 constitute a sparsely sampled 
image having red, green, and blue pixel values at neighboring pixel locations. 

10 In this description, "red", "green", and "blue" represent the primary 

spectral sensitivities of an image sensing device 10, as is well known in the art of 
image processing. The CFA interpolator 26 generates from the image signal 
output from the A/D converter 14 an interpolated image signal consisting of a 
pixel value corresponding to the color primaries for each photosite. For example, 

15 if a particular photosite is coated with a red filter, then the A/D converter 14 

outputs a red pixel value for that photosite since the red filter essentially blocks 
green and blue light from that particular photosite. The CFA interpolator 26 
calculates a green pixel value and blue pixel value for the corresponding photosite 
even though the corresponding photosite does not respond to green and blue light. 

20 Similarly, the CFA interpolator 26 calculates a green pixel value and a red pixel 
value corresponding to the blue photosites, as well as a red pixel value and blue 
pixel value corresponding to the green photosites. The operation of the CFA 
interpolator 26 can be combined with the DREFA processor 22. Conceptually, the 
CFA interpolator 26 and the DREFA processor 22 perform distinct operations and 

25 for the purpose of clarity are not combined in this embodiment. 

Generally, the CFA interpolator 26 operates by considering the 
pixel values of the corresponding photosite and the pixel values of associated 
surrounding photosites. While any commonly known interpolator may be used, a 
description of a preferred CFA interpolator is disclosed in. U.S. Patent No. 

30 5,652,621 issued July 29, 1997 to Adams et al. entitled Adaptive Color Plane 
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Interpolation in Single Sensor Color Electronic Camera, which is incorporated 
herein by reference. Adams et al. describe an apparatus for processing a digitized 
image signal obtained from an image sensor having color photosites aligned in 
rows and columns that generate at least three separate color values but only one 
5 color value for each photo site location, and a structure for interpolating color 
values for each photosite location so that it has three different color values. The 
apparatus generates an appropriate color value missing from a photosite location 
by the interpolation of an additional color value for such photosite locations from 
color values of different colors than the missing color value at nearby photosite 

10 locations. The apparatus also obtains Laplacian second-order values, gradient 
values and color difference bias values in at least two image directions from the 
pixel values corresponding to nearby photosites of the same column and row and 
selects a preferred orientation for the interpolation of the missing color value 
based upon a classifier developed from these values. Finally, the missing color 

1 5 pixel value from nearby multiple color pixel values is selected to agree with the 
preferred orientation. 

The sparsely sampled high resolution digital image output from the 
AID converter 14 is received by the DREFA processor 22 which expands the 
dynamic range of the image signal to create a modified image signal having an 

20 expanded dynamic range. In the preferred embodiment, the dynamic range of the 
image sensing device 10 is expanded by designing certain photosites of the image 
sensing device 10 to have a slow response. The arrangement of the slow response 
photosites with respect to the image sensing device 10 will be discussed in greater 
detail hereinbelow. The responses of the slow response photosites are slowed, or 

25 retarded, by altering the gain of the selected photosites, herein referred to as slow 
photosites. Altering the gain of a photosite is commonly practiced in the art of 
digital camera design and manufacture. 

With reference to Fig 3, it is a common practice in the art of image 
sensor manufacture to place resin lenslets 51 on top of each photosite. For 

30 example, particularly when the image sensing device 10 is an interline solid state 
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image sensing device, one such lenslet technique is described in U.S. Patent No. 
4,667,092 issued May 19, 1987 to Ishihara entitled Solid-State Image Device with 
Resin Lens and Resin Contact Layer, which is incorporated herein by reference. 
Ishihara discloses a solid-state image device which includes an image storage 
5 block having a block surface and a plurality of storage elements embedded along 
the block surface to store an image in the form of electric charge. An overlying 
layer is deposited to form an array of optical lenses in correspondence to the 
storage elements. An intermediate layer is laid between the block surface and the 
overlying layer. Incident light focuses through the lenses and the intermediate 

10 layer onto the storage elements. The intermediate layer serves as an adjusting 
layer for adjusting the focal length of the lenslets. 

Fig. 3 shows a cross section of an interline solid state image 
sensing device. Without the lenslets 51, the signal readout area associated with 
each photosensitive area 55 of a photosite makes it impossible to use the whole 

15 area of the semiconductor substrate as the photoelectric transducer area. The 

conventional solid-state image device does not effectively utilize all incident rays 
thereon and therefore has low sensitivity. The addition of a resin lenslet 51 on top 
of a photosite allows the incident rays of light to be focused on the photoactive 
areas of the photosite, thereby more effectively utilizing the incident rays of light 

20 and increasing the sensitivity of the photosite. Thus, by varying the size and/or 
efficiency of the lenslet 51, the sensitivity (gain) of the photosite may be easily 
altered. Thus, for interline devices and for CMOS sensors the preferred method of 
altering the gain of the photosite is by altering the lenslet 51 placed on top of the 
photosite. As shown in Fig. 3, the location 52 has no lenslet, and therefore fewer 

25 incident rays of light are incident on the photosensitive area. Alternatively, a 

lenslet could be manufactured at location 52 with a different radius, shape, size or 
material as compared with the lenslet 51, thereby structured to be less efficient at 
focusing incident rays of light onto the photosensitive area 55 than is the lenslet 
51. Those skilled in the art will recognize that if the lenslet 51 focuses 80% of the 

30 incident rays of light onto a photosensitive area 55 and the location 52 having no 
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lenslets (or alternatively slow lenslets) allows 20% of the incident rays of light 
onto a photosensitive area 55, then the photosite covered by lenslet 51 is 2 stops 
faster than the location 52. In this case, the lenslet 51 is used for fast photosites 
and no lenslet is used for slow photosites, as represented by location 52. 
5 With reference to Fig. 4 showing a cross section of a full frame 

image sensing device 10, in the case where the image sensing device 10 is a full 
frame device, light rays incident to the photosensitive area 55 of a photosite must 
pass through an aperture of a light shield, typically made from metal, which is 
shown in cross-section in Fig. 4 to comprise light blocking mask portions 54 and 

10 large and small apertures 56 and 57 interspersed among the light blocking 

portions. In the preferred embodiment, the gain of photosites may be altered by 
modifying the light blocking mask portion 54. The sensitivity of the photosite is 
then directly related to the aperture of the light blocking mask portion 54. For 
example, one photosite with an aperture 50% of the size of a second photosite's 

15 aperture will have a response of 50% compared to that on the second photosite. 
For example, a large aperture 56 of a light blocking mask portion 54 allows 80% 
of the light rays incident upon that photosite to pass through, but a small aperture 
57 allows only 20% of the incident light rays to pass. Those skilled in the art will 
recognize that the photosite with the large aperture 56 is 2 stops faster than a 

20 photosite with the small aperture 57. In this case, the large aperture 56 is used for 
fast photosites, and the small aperture 57 is used for the slow photosites. Thus, 
the aperture of the light blocking mask may be modified to adjust the response of 
the selected photosites. The Eastman Kodak Company makes full frame image 
sensing devices with a metal mask light shield that reduces the pixel active area of 

25 all pixels from about 80% to about 20% (for dithered scanner applications where 
the sensor is moved by 1/2 the pixel spacing horizontally and vertically, and 4 
pictures are taken). The technique thus involves utilizing such mask technology, 
but with different sized apertures, to provide an image sensor with a differential 
response to image light. 
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In the preferred embodiment, the response of the selected slow 
photosites is X% (where X<=\ 00) that of fast photosites for the same exposure, as 
shown graphically in Fig. 5. In this preferred embodiment, the selected photosites 
have a response that is slowed by two stops (-log Jf/100) relative to the fast 
5 photosites, resulting mX~ 25. Thus, the image sensing device 10 consists of 
multiple sets of photosites, fast photosites and slow photosites. The collection of 
the output responses of the fast photosites constitutes a sparsely sampled fast 
digital image, i.e. a sparsely sampled version of a scene sensed with the fast 
photosites. Likewise, the collection of the output responses of the slow photosites 

10 constitutes a sparsely sampled slow digital image, i.e. a sparsely sampled version 
of a scene sensed with the slow photosites. 

As another alternative, the responses of the selected slow photosites 
can be slowed by the use of a neutral filter coating the photosite. Fig. 6 shows a 
cross section of an image sensing device with a color filter array 53. Note that the 

1 5 color filter array 53a is red, 53b is green, 53c is red, and 53d is green. A layer of 
neutral filters 58 is contained above the color filter array 53, although the position 
of the layer of neutral filters 58 and the color filter array 53 does not matter. Note 
that the layer of neutral filters 58 only contains a neutral filter at the positions of 
selected photosites, as indicated by the neutral filter 59. In this case, the layer of 

20 neutral filters 58 is transparent or nearly transparent for fast photosites and 

contains a neutral filter 59 for slow photosites. For example, if the neutral filter 
59 consists of a material that allows X% transmission of light, then the response of 

that slow photosite will be slowed by - ^°§2^~~j stops relative to the response 

of the fast photosite. 

25 The DREFA processor 22 shown in Fig. 2 is described in more 

detail hereinbelow. The purpose of the DREFA processor 22 is to create a 
modified image signal with an increased dynamic range by processing the input 
image signal while accounting for the difference in photo response of the fast and 
slow photosites. Accordingly, the output of the DREFA processor 22 is a 

30 modified image signal having increased numerical dynamic range. This modified 
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image signal is then input to the CFA interpolator 26 for processing as previously 
described. 

It is not a requirement of the present invention that the A/D 
converter 14 and the DREFA processor 22 be directly connected. The DREFA 
5 processor 22 may reside in hardware or software in close proximity to the A/D 
converter 14 and image sensing device 10. For example, the DREFA processor 22 
could reside directly within a digital camera. However, the DREFA processor 22 
may also be remote from the image sensing device 10. For example, the image 
signal output from the A/D converter 14 can be transmitted (after compression) 

10 via a wire or wireless connection to a personal computing device, printer, or 

remote server to apply to operation of the DREFA processor 22. Transmission of 
the image signal may also include file transfer protocol or email. 

In the preferred embodiment, 50% of the photosites of the image 
sensing device 10 are selected to have slow response. Those skilled in the art will 

15 recognize that the relative percentages of slow and fast pixels is not critical, and 
that the advantages of the present invention can be achieved with various relative 
percentages of fast and slow photosites. In the case of an image sensing device 10 
in which all photosites have approximately equivalent spectral sensitivity (i.e. a 
panchromatic image sensing device), Fig. 7A shows an arrangement of the slow 

20 photosites that will result in approximately 50% of all the photosites of the image 
sensing device 10 being of slow response. The photosites 28 with slow response 
are marked with an asterisk (*), while the photosites 30 having fast response are 
blank. A sparsely sampled image was previously defined as an image that was 
captured with an image sensing device 10 having a color filter array. According to 

25 the present invention, the term sparsely sampled is also intended to refer to an 

image produced by an image sensing device such as that shown in Fig. 7 A where 
the fast and slow photosites are interspersed. Additionally, an image sensing 
device 10 such as shown in Fig. 7 A having fast photosites with a predetermined 
response to light exposure interspersed with slow photosites having a slower 



-14- 



response to the same light exposure is a sparsely sampled extended dynamic range 
image sensing device. 

Fig. 7B shows an arrangement for a color image sensing device 10 
wherein 50% of each photosite type (red, green, or blue sensitive) has slow 
5 response. For example, the photosites 32, 34, and 36 are red, green and blue 

photosites, respectively, having slow responses; the photosites 38, 40, and 42 are 
red, green and blue photosites, respectively, having fast responses. Note that the 
image sensing device 10 is also a sparsely sampled extended dynamic range image 
sensing device according to the previous definition. 

1 0 Note that Figs. 7A and 7B imply a regular pattern for the location 

of the slow photosites. While it is preferable that the slow photosites are arranged 
in a regular pattern, it is by no means necessary. The slow photosites could be 
arranged randomly or semi-randomly over the surface of the image sensing device 
10, and their location would be stored in some place accessible to the DREFA 

15 processor 22. 

Referring to Fig. 5, the response of a fast photosite to a certain 
exposure and the response of a slow photosite to the same exposure are shown. 
Note that if a level of noise n is superimposed on the response, it can easily be 
seen that the fast photosite will yield a valid signal with lower exposures 
20 (beginning at exposure level E) than will the slow photosite (which yields valid 

signal beginning at .) Alternatively, data from the slow photosite will be 

valid for higher exposure levels (up to signal level of -^-E2 S , where S is the 

inherent dynamic range of a single photosite, typically S may be about 5 stops) 
than would the fast photosite (which produces valid response up to an exposure of 

25 E2 S ). Note that both the fast photosite and the slow photosite have the same 

range of response in stops of exposure (S), but the response of the slow photosites 

is preferably - *°§2^Y~~j stops slower than the fast photosites, as shown in Fig. 

5. It is preferred that the responses of the fast and slow photosites overlap with 
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respect to exposure. That is, it is preferred that - log2^j^J< -S 1 . The overall 

dynamic range of the image sensing device 10, considering both fast and slow 

photosites, is S - log 2 ^ ~^ j • ^ n tne case °^ tne P re f erre d embodiment, where S = 

5 andX= 25, the overall effective dynamic range of the image sensing device 10 
5 is 7 stops of exposure. 

The DREFA processor 22 may be utilized to extend the overall 
dynamic range of the digital images produced with the present invention by using 
the pixel values corresponding to slow photosites to reconstruct the image signals 
in regions corresponding to very high exposures. Likewise, the DREFA processor 

10 22 also uses the pixel values corresponding to photosites with fast response to 
reconstruct the image signal corresponding to very low exposures. 

Fig. 8 shows a block diagram of the DREFA processor 22. The 
sparsely sampled high resolution digital image, which is a logarithmic image 
signal b(x,y) output from the AfD converter 14, is passed to the slow pixel 

1 5 compensator 44. The purpose of the slow pixel compensator 44 is to compensate 
the image signal corresponding to slow photosites by accounting for the offset in 
response by X stops. Alternatively, the fast pixels can be equalized to the slow 
pixels by adjusting the fast pixels in the opposite direction. In the preferred 
embodiment, the image signal corresponding to the slow photosites are 

20 incremented by the quantity -cvs log (X/100), where cvs is the number of code 
values per stop of exposure. In the preferred embodiment, the quantity cvs is 5 1 . 
Alternatively, if the image signal input to the slow pixel compensator 44 is 
linearly related to exposure (rather than logarithmically), then the slow pixel 
compensator 44 scales the image signal corresponding to the slow photosites by a 

25 factor of 1 00/X Note that it is assumed that the locations of the slow photosites 
are known to the slow pixel compensator 44. The output of the slow pixel 
compensator 44 is an image signal i(x,y) that has been compensated at the 
locations of corresponding to slow photosites for the difference between the slow 
photosite response in relation to the fast photosite response. At the locations 
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corresponding to fast photosites, the value of the image signal b(x,y) output from 
the A/D converter 14 is identical to the value of the image signal i{x,y) output 
from the slow pixel compensator 44. Note that the image signal i(x,y) is not 
limited to an 8 bit range. In the preferred embodiment, the value of i(x,y) ranges 
from 0 to 357 (i.e. 9 bits). 

Next, the image signal i{x,y) output from the slow pixel 
compensator 44 is input to a slow pixel thresholder 46. The purpose of the slow 
pixel thresholder 46 is to determine slow pixel values that are of low quality due 
to a photosite not receiving enough photons to produce a valid signal. The pixel 
value at these (x,y) locations is then replaced in processing performed by the 
signal extender 50 by calculating a new pixel value based upon nearby fast pixel 
values. All slow pixel values which are less than a predetermined threshold are 
considered to be problem pixel values. In the case of the slow pixel values, this 
predetermined threshold is referred to as the low exposure response threshold. 
Thus, a pixel value i(x,y) is considered to be a problem if it is a slow photosite and 
if: 

i(x,y) < Tj (2) 
where T / is predetermined. In the preferred embodiment, the value of Tj is given 
by 

TJ = -cv slo g2 g), (3 ) 

which in the preferred embodiment is set to a value of 102. Note that the 
threshold Tj may be dependent upon the color sensitivity of the photosite at 
location (x,y). Slow pixel values that are problems are referred to as noise pixels, 
since the value of i(x,y) is not sufficiently above the noise level of the image 
sensing device to be useful. 

Likewise, the image signal i(x,y) output from the slow pixel 
compensator 44 is input to a fast pixel thresholder 48. The purpose of the fast 
pixel thresholder 48 is to determine fast pixels that are of low quality. The pixel 
values at these locations is then replaced by calculating a new pixel value based 
upon nearby slow pixel values in processing performed by the signal extender 50, 
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which will be described in detail hereinbelow. All fast pixel values that are 
greater than a predetermined threshold value are considered to be problem pixels. 
In the case of the fast pixels, this predetermined threshold used for the purpose of 
detecting problem fast pixels is referred to as the high exposure response 
5 threshold. Thus, a fast pixel value i(x,y) is considered to be a problem if: 

i(x,y)>T 2 (4) 
where J? is a predetermined threshold. In the preferred embodiment, the value of 
T2 is set to a value of 254. Note that the threshold T 2 may be dependent upon the 
color of the photosite at location (x,y). Fast photosites that are problem locations 
10 are referred to as saturated pixels, since the value of i{x,y) is as high as possible at 
these locations. 

The (x,y) locations of the problem slow pixels determined by the 
slow pixel thresholder 46 and the (x,y) locations of the problem fast pixels 
determined by the fast pixel thresholder 48 are input to the signal extender 50. In 

15 addition, the image signal i(x,y) output from the slow pixel compensator 44 is also 
input to the signal extender 50. The purpose of the signal extender 50 is to replace 
the image signal i(x,y) values at problem locations (x,y) with estimates of the 
signal herein referred to as replacement values, had the inherent dynamic range of 
each photosite of the image sensing device 10 been greater. If the problem 

20 location is coincident with a slow photosite, then the replacement value is 
calculated from neighboring image signal pixel values coincident with fast 
photosites. In this embodiment, the term "neighboring" refers to a certain spatial 
distance. In the preferred embodiment, the photosites neighboring a selected 
photosite are those photosites within a distance of 2 photosites of the selected 

25 photosite. Likewise, if the problem location is coincident with a fast photosite, 
then the replacement value is calculated from neighboring image signal values 
coincident with slow photosites. In the preferred embodiment, the color of the 
photosite at the problem photosite is also taken into account. The replacement 
value for any problem location is preferably determined only by the signal 

30 originating from neighboring photosites of the same color. The output of the 
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signal extender 50 is an image signal i'{x,y) having a dynamic range as if captured 
by an image sensing device 10 having photosites with inherent dynamic range of 



rather than the actual inherent dynamic range of S for each photosite of the image 
5 sensing device 10. Note that for all (x,y) locations that are not problem locations, 
the value of i'(x,y) is equivalent to i(x,y). 



50 for the Bayer CFA pattern shown in Fig. 7B, if location (x,y) is a problem 
location, and (x,y) is the location that corresponds to a green photosite (such as 

1 0 photosite 34 in Fig. 7B), then the replacement value i'{x,y) for the image signal 
i(x,y) is calculated in the following manner: 

i\x,y) = 0.25 *[i(x-l,y-l) + i(x+\,y-l) + i(x-l,y+l) + i(x+l,y+l)] (6) 
Note that signal values that the calculation of i'(x,y) is dependent upon, are 
expected to comply with certain requirements. For example, suppose that (x, y) is 

1 5 a problem location and (x, y) corresponds to a green photosite with slow response. 
Then the signal levels of neighboring photosites are used to calculate replacement 
value i\x,y). However, this assumes that the signal values of each of the 
neighboring photosites are also less than T3. In the preferred embodiment, T3 = 
Tj. For each neighboring photosite that this is not the case, that signal level is 

20 omitted from the calculation of the replacement value i'(x,y). For example, if i(x- 
1, y-1) > T3, then the value i\x,y) is calculated with the following formula: 



Generally, the interpolation scheme for determining a replacement value at 
problem location (x,y), where the location (x,y) corresponds to a green photosite 
25 which is also a fast photosite on a image sensing device having a Bayer pattern 
filter array is given with the following equation: 




As an example of the processing performed by the signal extender 



i'(x,y) = 1/3 *[i(x+l,y-l) + i(x-\,y+l) + i(x+l,y+\)] 



(7) 



i'(x,y) = 



X E i ( x + j>y +k ) w ( x + j>y+k) 
j=-uk=-i,i 



X £w(x + j,y + k) 



(8) 



j=-l,lk=-l,i 
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™, ■ i ^ f 1 i(x + j,y + k) > 
W(x + j,y + k) = <^ 

[0 otherwise 



(9) 

Note that the same equation is applied to determine the replacement value if the 
problem location corresponds to a green photosite which is also a slow photosite. 
However, in this case: 

W(x + j y + k) - j 1 ^ X + J ' y + k ) < T 4 
[0 otherwise ' 

where in the preferred embodiment, T4 = r?. 

As another example, also in connection with the Bayer CFA pattern 

shown in Fig. 7B, if location i(x,y) is a problem photosite and (x,y) corresponds to 

a location of a red or blue photosite, then the replacement value i'(x,y) for the 

image signal i(x,y) is calculated in the following manner: 

i'(x,y) = 0.25 *[i(x-2,y) + i(x+2,y) + i(x,y+2) + i(x,y-2)]. (1 1) 

When location (x,y) corresponds to a red or blue photosite and is also a fast 

photosite, the equation for determining the replacement value i'(x,y) may be 

generalized as follows: 

S Z f ( x + j' y + k ) w ( x + j» y + k) 

■ u . j=-2,0,2k=-2,0,2 

,(x ' y)= I Zw (x+ , y+k) — < 12 > 

j=-2,0,2k = -2,0,2 



W(x + jy + k)-{ 1 i( x + j»y + k) >T 3 

[0 otherwise ^ 13 ^ 

Note that in this case, either/ or k must be 0, but j and k are never both zero. Note 

also that the same equation is applied to determine the replacement value if the 

problem location corresponds to a red or blue photosite which is also a slow 

photosite. However, in this case 

• [l i(x + j,y + k) <T 4 

W(X+J ' y + k) = { 0 otherwise • ™ 

where in the preferred embodiment, T4 = T^. 
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The interpolation scheme described above for the purpose of 
generating an image signal with an extended dynamic range from more than one 
sparsely sampled image signal may be modified by those skilled in the art. 
However, many such modifications to the above interpolation scheme may be 
5 derived and should not be considered as significant deviations of the present 
invention. 

Those skilled in the art will recognize that the above interpolation 
scheme performed by the signal extender is a lowpass filter, which is well known 
in the art. Typically, the application of a lowpass filter to an image signal has a 

10 similar effect to reducing the resolution of the image signal. Thus, the processing 
performed by the DREFA processor 22 is a method by which the spatial resolution 
of the image sensing device 10 may be traded for dynamic range of the image 
sensing device 10. Indeed, those areas of an image where the interpolation 
scheme is implemented to increase the dynamic range of the signal appear 

15 noticeably softer (less sharp) than the image would have if that same area of the 
image had been captured by the image sensing device in such a fashion that no 
"problem locations" occur (as defined by the slow pixel thresholder 46 and the 
fast pixel thresholder 48). 

The paxelization module 220 shown in Fig. 2 is illustrated in more 

20 detail in Fig. 9. The purpose of the paxelization module 220 is to create a 
paxelized digital image for the purpose of analysis by algorithms in order to 
determine density and color balance and determine a desired tone scale function 
by the enhancement processor 240 as will be described in detail hereinbelow. The 
paxelization module 220 receives an image signal from the AID converter 14. The 

25 image signal is received by the pixel type separator 222. The purpose of the pixel 
type separator 222 is to allow access to each type of pixel separately. In the 
present invention, the photosites are of two types, fast or slow as shown in Fig. 7 A 
in the case of a monochromatic image sensing device 10. Furthermore in the case 
of the color image sensing device 10 as shown in Fig. 7B, within each category, 

30 the photosite may also be red, green, or blue. (Those skilled in the art will 
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recognize that many other color and speed combinations are possible.) Thus there 
are a total of six types of photosites. In this embodiment, the pixel type separator 
222 separately outputs all the fast pixel values and all the slow pixel values. The 
paxelization engine 224 receives the slow pixel values and performs a numerical 
averaging spatial filtering technique which results in a paxelized digital image. In 
the preferred embodiment, the paxelization engine 224 utilizes block averaging 
with block size Nx Nin order to create the paxelized digital image which is 
output from the paxelization module 220. For example, if a block size of 7V = 32 
pixels is used by the paxelization engine 224, then when the sparsely sampled 
high resolution digital image has a resolution of 1024 x 1536, then the paxelized 
digital image will have a resolution of 32 x 48 as illustrated in Fig. 10. Typically 
the value of Wis selected such that the paxelized digital image is of small size, 
such as is used by color and tone analysis algorithms. Each pixel of the paxelized 
digital image corresponds to a TV x AT block of the sparsely sampled high 
resolution digital image. Each pixel value of the paxelized digital image has 
associated pixel value for each color of the digital image, typically red, green, and 
blue. The red pixel values of the paxelized digital image are determined by the 
paxelization engine 224 by averaging all of the red pixel values within the 
corresponding Nx N block of the sparsely sampled high resolution digital image. 
Notice that since in the preferred embodiment the pixel type separator 222 outputs 
only slow pixel values, each pixel of the paxelized digital image will be calculated 
exclusively with slow pixel values. A similar process is followed for the green 
and the blue pixel values. 

It is important to note that the action taken by pixel type separator 
222 does not have to rearrange the storage of the pixel data in computer memory. 
The present invention implements the pixel type separator 222 as a pointer 
addressing scheme to the storage of the pixel data in computer memory. Thus the 
most important aspect of the pixel type separator 222 is capability of indexing 
pixel data corresponding to the fast photosites and slow photosites as a separate 
entities. 
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An alternative embodiment of the pixel type separator 222 
generates a paxelized image having only a single pixel value at each pixel 
location. This pixel value can be thought of as a luminance pixel value. The 
luminance pixel values of the paxelized digital image can be obtained by using 
only the slow green pixel values within the corresponding Nx TV block of the 
sparsely sampled high resolution digital image. Alternatively, luminance pixel 
values may be derived by using all of the slow pixel values within the 
corresponding Nx TV block of the sparsely sampled high resolution digital image. 
As a further alternative embodiment illustrated in Fig. 2, the paxelization module 
220 may use similar techniques to generate a paxelized digital image from the 
image signal output from the DREFA processor 22 or from the full resolution 
digital image output from the CFA interpolator 26. 

The low resolution module 232 shown in Fig. 2 creates a low 
resolution digital image for the purpose of allowing a user to interactively 
determine balance and tone scale function modifications by allowing the user to 
specify desired tone scale adjustments. The user inputs these desired tone scale 
adjustments with the user selection device 62, shown in Fig. 1. The operation of 
the low resolution module 232 (see Fig. 2) is similar to the paxelization module 
220 except that the factor of TV is generally selected so the low resolution image is 
appropriate for the display device 64. Generally, the factor of N for the low 
resolution module 232 is greater than the factor of N for the paxelization module 
220. In the preferred embodiment, the factor of TV used to generate the low 
resolution digital image is 8. If the display device 64 is a color display device, 
then the low resolution digital image should have more than one color pixel value 
for each pixel location. Alternatively, if the display device 64 is not a color 
display device, then the low resolution digital image may have only a single 
luminance pixel value as previously described. 

The enhancement module 240 shown in Fig. 2 is illustrated in more 
detail in Fig. 11. The full resolution digital image output from the CFA 
interpolator 26 is input to the enhancement processor. In an alternative 
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embodiment, the position of the enhancement processor 24 may be placed in from 
of the CFA interpolator 26 or the DREFA processor 22. 

The luminance-chrominance module 320 1 receives the full 
resolution digital image and generates a full resolution LCC digital image. The 
5 luminance-chrominance module 320 2 receives the low resolution digital image 
and generates a low resolution LCC digital image. In applications such as 
monochrome imaging, the luminance chrominance module 320 may be omitted. 

The scene balance module 310 receives the paxelized digital image 
and calculates balance parameters which relate to an overall in color and 

1 0 brightness change to be imparted to final processed digital image. The paxelized 
digital image and the balance parameters are received by the tone scale function 
generator 330 which calculates a lightness tone scale function, i.e. a single valued 
function used for transforming pixel values. The tone scale function applicator 
340 2 receives the lightness tone scale function and the low resolution LCC digital 

1 5 image and applies the lightness tone scale function to the low resolution LCC 

digital image to produce a tone scale modified low resolution digital image. The 
tone scale modified low resolution digital image represents an extended dynamic 
range digital image which has been balanced for brightness and color and had its 
tone scale adjusted to improve shadow and highlight detail. The tone scale 

20 modified low resolution digital image is received by the rendering module 350 
which produces a low resolution rendered digital image. The process of 
converting the color values captured by an image sensing device to those 
appropriate for display on a particular output device is often referred to as 
"rendering". The low resolution rendered digital image is then displayed on a 

25 display device 64 for the purpose of gathering user feedback. The user inputs 

desired changes to the tone scale function using a user selection device 62 such as 
a button, mouse, touch screen, slider, voice commands, or other methods of 
inputting the desired tone scale adjustments. As an example of the desired tone 
scale adjustments, the user indicates by way of the user selection device 62 that 

30 the highlights of the image on the display device 64 should be darkened. The 
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scene balance module 310 and the tone scale function generator 330 then generate 
modified balances and tone scale functions, respectively. Making image balance 
and tone scale dependent on user interaction is well known in the art. For 
example, U.S. Patent No. 5,012,333 issued April 30, 1991 to Lee et al. entitled 
Interactive Dynamic Range Adjustment System for Printing Digital Images 
describes an interactive method for setting then applying an image dependent tone 
scale function. The tone scale function applicator 340 2 receives the modified tone 
scale function and the low resolution LCC digital image and applies the modified 
tone scale function to the low resolution LCC digital image to produce a tone 
scale modified low resolution digital image. Note that several methods exist for 
applying a tone scale function to a digital image and will be described in greater 
detail hereinbelow. After rendering by the rendering module 350, the low 
resolution rendered digital image is displayed to the display device 64 in order to 
allow the user to make further adjustments with the user selection device 62. This 
process may repeat itself until the user has no further adjustments. In that case, 
the tone scale function applicator 340, applies the tone scale function (or the 
modified tone scale function in the case where the user makes no adjustments) to 
the full resolution LCC digital image in order to produce a tone sale adjusted full 
resolution LCC digital image. The RGB conversion module 360 converts this 
LCC digital image back to an image represented with red, green, and blue pixel 
values to produce a tone scale adjusted full resolution digital image. 

In an alternative embodiment, in some applications such as high 
speed applications, the display device 64 and user selection device 62 may be 
omitted. In this case, the low resolution digital image need not be created, and the 
tone scale function output from the tone scale function generator 330 is applied to 
the full resolution LCC digital image by the tone scale function applicator 340,. 

The luminance-chrominance module 320, is used to generate a full 
resolution LCC digital image from the full resolution digital image. The full 
resolution digital image includes red, green, and blue digital image channels. 
Each digital image channel contains the same number of pixels. Let Rjj, Gjj, and 
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By refer to the pixel values corresponding to the red, green, and blue digital image 
channels located at the i ft row and j th column. Let L[j, Cly, and C2jj refer to the 
transformed pixel values of the modified digital image. The 3 by 3 matrix 
transformation relating the input and output pixel values is as follows: 

L ij = -en Rij + t 12 Gij + Ti3 By (15) 

C1 ij = m R ij + T 22 Gij + x 23 Bjj 

C2 ij = *31 R ij + ^32 Gij + x 33 Bjj 
where the x mn terms are the coefficients of the 3 by 3 LCC matrix denoted by [r]. 
The constants employed by the present invention for tj j , x\2 and T13 are 0.333, 
0.333 and 0.333 respectively. It is important to note that the present invention 
may be practiced with other luminance chrominance transformations and still 
yield good results. For example, a matrix with -q 1 , iyi and T 13 values set to 

0. 30, 0.59, and 0.1 1 respectively also works well. The calculated values CI and 
C2 are examples of digital color difference values. The luminance-chrominance 
module 32 0 2 is also used to generate, in identical manner, a low resolution LCC 
digital image. 

The paxelized digital image output from the paxelization module 
220 is input to the scene balance module 310. The present invention may be 
practiced with any scene balance module 310 such as the one described in U.S. 
Patent No. 4,945,406 issued July 31, 1990 to Cok entitled Apparatus and 
Accompanying Methods for Achieving Automatic Color Balancing in a Film to 
Video Transfer System. The scene balance module calculates balance parameters, 

1. e. pixel values corresponding to the pixel values of theoretical 18% gray card, 
based on the pixel values of the paxelized digital image. In the method taught by 
Cok, statistical parameters are calculated from the pixels of the paxelized digital 
image such as pixel minimum, maximum and average values. These statistical 
parameters are calculated for different regions of the paxelized digital image and 
for three color digital image channels (L, CI and C2). The statistical parameters 
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are combined in a multilinear equation with predetermined coefficients to predict 
three balance parameters, i.e. an L channel balance parameter relating to the 
overall image brightness and two chrominance balance parameters for the CI and 
C2 digital image channels which relate to the overall color balance of the image. 
5 Note that these color balance values, although expressed in terms of L, CI and C2 
may be easily converted to other color spaces (e.g. red, green, and blue) by 
techniques commonly known to those in the field of color science. 

The tone scale function generator 330 receives the paxelized digital 
image and calculates a tone scale function, i.e. a single valued mathematical 

1 0 equation or transformation that has a single output value corresponding to each 

input value. The present invention implements the lightness tone scale function as 
a look-up-table for computation efficiency. The present invention may be 
practiced with a variety of methods which generate tone scale functions. The 
preferred embodiment of the present invention uses the methods disclosed in U.S. 

15 Patent No. 4,731,671 issued March 15, 1988 to Alkofer entitled Contrast 
Adjustment in Digital Image Processing Method Employing Histogram 
Normalization and U.S. Patent No. 5,822,453, referenced above. These methods 
are employed by the present invention to produce two individual tone scale 
functions. These two tone scale functions are then cascaded into single lightness 

20 tone scale function which is used to adjust the brightness and contrast of the LCC 
digital image. 

In U.S. Patent No. 5,822,453, Lee et al. disclose a method of 
calculating a tone scale function using the pixel values of a digital image 
involving estimating the scene contrast from the digital image. The method taught 

25 by Lee et al. involves calculating a Laplacian filtered version of the digital image, 
forming a histogram of the Laplacian signal, determining from the Laplacian 
histogram, two threshold values which when applied to the Laplacian signal 
substantially eliminate uniform areas; sampling pixels from the digital image 
which are based on the thresholds; forming a histogram from the sampled pixels; 

30 computing a standard deviation of the sampled histogram; and estimating contrast 
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of the digital image by comparing the computed standard deviation with a 
predetermined contrast for determining contrast of the input image in relationship 
with the predetermined contrast. The method described by Lee and Kwon is used 
to calculate a first tone scale function. 
5 In U.S. Patent No. 4,731,671, Alkofer discloses a method of 

calculating a tone scale function using the pixel values of digital image based on 
normalizing the histogram of a digital image. This method involves determining 
the contrast of the digital image by calculating the standard deviation of a sample 
of pixel values. The second tone scale function is calculated by normalizing a 

10 histogram of the sample of pixels values. The sample of pixel values is selected 
from one of a plurality of samples of pixel values corresponding to a plurality of 
contrast intervals based upon the shape of the histogram of the selected sample of 
pixel values. To facilitate the adjustment of contrast, the tone scale function is 
constructed to produce values in units of a standard normal variate Z. These Z 

15 values are then multiplied by a constant which is a function of the standard 

deviation of the sample of pixel values to determine the contrast of the processed 
digital image. The present invention uses the L (luminance) digital image channel 
of the LCC paxelized digital image as a base for analysis of the second tone scale 
function. 

20 The first and second tone scale functions are combined into a 

lightness tone scale function using the mathematical formula: 

LUT f [j] = LUT, [ LUT 2 [j] ] (16) 
where LUT 2 represents the second tone scale function, LUTj represents the first 
tone scale function, and LUT f represents the lightness tone scale function. The j 
25 variable represent the index of pixel values of the digital image to be processed. 
The lightness tone scale function LUT f is calculated by evaluating the expression 
of equation (16) for the range of possible pixel values. 

The lightness tone scale function LUT f and the LCC digital image 
are received by the tone scale function applicator 340, and 340 2 . The present 
30 invention applies the lightness tone scale function to the luminance channel of the 
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ftill resolution LCC digital image and the low resolution LCC digital image 
respectively to adjust the brightness and contrast attributes of the source digital 
image. Applying the lightness tone scale function directly to the L (luminance) 
channel of the LCC digital image is the fasted method. However, the present 
5 invention may be practiced with other methods of applying the lightness tone 
scale function to the pixels of a digital image. For example, an alternative 
embodiment is described in U.S. Serial No. 09/163,645 filed September 30, 1998 
by Gallagher et al. Alternatively, commonly assigned U.S. Patent No. 5,012,333 
also describes a method of applying a tone scale function to a digital image. 

1 0 These methods involve spatially filtering the luminance digital image channel 
resulting in two spatial frequency components (high and low components), 
applying the tone scale function to the low spatial frequency component, and 
combining the tone scale modified low spatial frequency component with the high 
spatial frequency component. The resulting processed digital image has enhanced 

1 5 brightness and contrast attributes with improved spatial detail content. Both of 
these methods utilize spatial filters in order to apply the tone scale function. 

The RGB conversion module 360 receives tone sale adjusted full 
resolution LCC digital image and transforms the luminance-chrominance 
representation back to a red-green-blue channel representation with the application 

20 of a linear 3 by 3 matrix transform. The resulting digital image channels of the 
tone scale adjusted full resolution digital image have the same color metric 
representation as the source digital image. The transformation generates new 
pixel values as linear combinations of the input color pixel values. 

Let Ljj, Cljj, and C2y refer to the pixel values corresponding to the 

25 luminance and two chrominance digital image channels located at the i th row and 
j th column. Let R'jj, G'jj, and B'jj refer to the transformed pixel values of the 
modified digital image. The 3 by 3 matrix transformation relating the input and 
output pixel values is as follows: 

R'ij = miLij + rifely + r| 13 C2ij (17) 
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G 'ij = 121 L ij + 122 Cly + 7123 C2y 

B 'ij = 131 Lij + -H32 Clij + T133 C2y 
where the r| mn terms are the coefficients of the 3 by 3 matrix transformation. 

The preferred embodiment of the present invention constructs the 
rgb conversion matrix denoted by [n] above as the inverse of the lcc matrix 
denoted by [t] corresponding to the luminance-chrominance module 320. This is 
mathematically represented in matrix notation in equation (18). 

[1] = W" 1 (18) 

The rendering module 350 shown in Fig. 1 1 receives tone scale 
modified low resolution digital image, applies a transform to the pixel values, and 
produces a low resolution rendered digital image that is directly viewable on a 
display device. Two transforms are used to prepare the tone scale modified low 
resolution digital image for direct viewing. The first transform is a 3x3 color 
matrix transformation which is applied to the color pixels of the tone scale 
modified low resolution digital image. The color matrix transformation accounts 
for the difference between the spectral sensitivities of the color photosites of the 
image sensing device 10 and the spectral characteristics of the display device 64. 
The method described above employed by the present invention is similar to the 
method taught and disclosed in U.S. Patent No. 5,189,51 1 issued February 23, 
1993 to Parulski et al. entitled Metho d and Apparatus for Improving the Color 
Rendition of Hardcopy Images from Electronic Cameras. The second transform 
involves the application of a tone scale function which maps the extended 
dynamic range pixel values of the tone scale modified low resolution digital image 
to pixel values compatible with typical viewing devices. The present invention 
uses a similar method to the one described in U.S. Patent No. 5,300,381 issued 
April 5, 1994 to Buhr et al. entitled Color Image Reproduction of Scenes with 
Preferential Tone Mapping. Buhr describes a method of calculating a rendering 
tone scale function with preferential tone mapping in which the contrast of the 
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tone scale function is greatest for midtone pixel intensities and has gracefully 
lower contrast for shadow and highlight pixel intensities. This rendering tone 
scale function is combined with a gamma mapping tone scale function to calculate 
a system tone scale function. The gamma mapping tone scale function 
5 compensates for the inherent intensity response of typical viewing devices. The 
system tone scale function is cascaded with the rendering tone scale function and 
applied, in the form of a look-up-table, to the pixels of the tone scale modified low 
resolution digital image resulting in a low resolution rendered digital image. 

For digital camera applications, a display device 64 such as an 

1 0 LCD screen is used to view digital images produced by the digital image 

processor 200. For other digital imaging applications the user can also view 
digital images on a display device 64. The user views the image presented on the 
LCD display device and can make selections with regard to the brightness, color 
and tone. The changes to the brightness and color are input to the scene balance 

1 5 module 310 as shown in Fig. 1 1 . The present invention uses the modifications 
selected by the user to make additive changes to the balance parameters. Those 
skilled in the art of color balance technology will recognize that the present 
invention can be practiced with a wide variety of different methods for achieving 
color balance. 

20 The invention has been described in detail with particular reference 

to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention 
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PARTS LIST 
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6 


optical lowpass filter 


10 


image sensing device 


14 


A/D converter 


22 


DREFA processor 


26 


CFA interpolator 


28 


slow photosite 


30 


fast photosite 


32 


red slow photosite 


34 


green slow photosite 


36 


blue slow photosite 


38 


red fast photosite 


40 


green fast photosite 


42 


blue fast photosite 


44 


slow pixel compensator 


46 


slow pixel thresholder 


48 


fast pixel thresholder 


50 


signal extender 


51 


standard lenslet 


52 


location 



53a-d color filter array 

54 light blocking mask portion 

55 photosensitive area 

56 large aperture 

57 small aperture 

58 neutral density filter layer 

59 neutral density filter 
62 user selection device 
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64 display device 

200 digital image processor 

220 paxelization module 

222 pixel type separator 

224 paxelization engine 

232 low resolution module 

240 enhancement processor 

310 scene balance module 

320, luminance-chrominance module 

320 2 luminance-chrominance module 

330 ton scale function generator 

340; tone scale applicator 

340 2 tone scale applicator 

350 rendering module 

360 RGB conversion module 
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WHAT IS CLAIMED IS: 

1 . A method of generating a tone scale function for a sparsely 
sampled extended dynamic range digital image, comprising the steps of: 

a) providing a sparsely sampled extended dynamic range image 
sensing device having fast photosites with a predetermined response to light 
exposure interspersed with slow photosites with a slower response to the same 
light exposure; 

b) using the image sensing device to produce a sparsely sampled 
high resolution digital image having fast pixel values produced by the fast 
photosites and slow pixel values produced by the slow photosites; and 

c) generating the tone scale function using only slow pixel values 
from the sparsely sampled high resolution digital image. 

2. The method claimed in claim 1, further comprising the step of: 

d) applying the tone scale function to the sparsely sampled high 
resolution digital image to produce a tone scale adjusted sparsely sampled high 
resolution digital image. 

3. The method claimed in claim 2, further comprising the step of: 

e) generating a full resolution digital image from the tone scale 
adjusted sparsely sampled high resolution digital image to produce a tone scale 
adjusted full resolution digital image. 

4. The method claimed in claim 1, further comprising the steps of: 

d) generating a full resolution digital image from the sparsely 
sampled high resolution digital image; and 

e) applying the tone scale function to the full resolution digital 
image to produce a tone scale adjusted full resolution digital image. 
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5. The method claimed in claim 1, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein the slow pixel values from only one 
color of photosite are used to generate the tone scale function. 

6. The method claimed in claim 5, wherein the different colors are 
red, green, and blue and wherein only green pixel values are used to generate the 
tone scale function. 

7. The method claimed in claim 1, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and further comprising the step of: 

d) constructing slow luminance pixel values from the different 
colored slow pixel values; and 

e) using the slow luminance pixel values to generate the tone scale 

function. 

8. The method claimed in claim 1, further comprising the steps of: 

d) forming a paxelized digital image from the slow pixel values; 

and 

e) using the paxelized digital image to generate the tone scale 

function. 

9. The method claimed in claim 4, further comprising the step of 
applying a spatial filter to the full resolution digital image. 

10. The method claimed in claim 1, further comprising the steps 

of: 

d) generating a low resolution digital image from the sparsely 
sampled high resolution digital image; 
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e) applying the tone scale function to the low resolution digital 
image to form a tone scale modified low resolution digital image; 

f) displaying the tone scale modified low resolution digital image; 

g) specifying a desired tone scale adjustment of modified low 
resolution digital image; and 

h) employing the specified tone scale adjustment to generate an 
adjusted tone scale function. 

1 1 . The method claimed in claim 1, wherein the method is 
performed in a digital camera. 

12. The method claimed in claim 1, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein a tone scale function is generated for 
each color. 

13. The method claimed in claim 12, further comprising the steps 

of: 

d) generating a full resolution digital image from the sparsely 
sampled high resolution digital image; and 

e) applying the tone scale functions to the full resolution digital 
image to produce a tone scale adjusted full resolution digital image. 

14. A method of generating a tone scale function for a sparsely 
sampled extended dynamic range digital image, comprising the steps of: 

a) providing a sparsely sampled extended dynamic range image 
sensing device having fast photosites with a predetermined response to light 
exposure interspersed with slow photosites with a slower response to the same 
light exposure; 
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b) using the image sensing device to produce a sparsely sampled 
high resolution digital image having fast pixel values produced by the fast 
photosites and slow pixel values produced by the slow photosites; 

c) analyzing the fast pixel values for saturation; and 

d) if saturation is present in the fast pixel values, generating the 
tone scale function using only slow pixel values from the sparsely sampled high 
resolution digital image, otherwise, using only fast pixel values. 

15. The method claimed in claim 14, further comprising the step 

of: 

d) applying the tone scale function to the sparsely sampled high 
resolution digital image to produce a tone scale adjusted sparsely sampled high 
resolution digital image. 

16. The method claimed in claim 15, further comprising the step 

of: 

e) generating a full resolution digital image from the tone scale 
adjusted sparsely sampled high resolution digital image to produce a tone scale 
adjusted full resolution digital image. 

17. The method claimed in claim 14, further comprising the steps 

of: 

d) generating a full resolution digital image from the sparsely 
sampled high resolution digital image; and 

e) applying the tone scale function to the full resolution digital 
image to produce a tone scale adjusted full resolution digital image. 



18. The method claimed in claim 14, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
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producing color pixel values and wherein the pixel values from only one color of 
photosite are used to generate the tone scale function. 

19. The method claimed in claim 18, wherein the different colors 
are red, green, and blue and wherein only green pixel values are used to generate 
the tone scale function. 

20. The method claimed in claim 14, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and further comprising the step of: 

d) constructing luminance pixel values from the different colored 
pixel values, and 

e) using the luminance pixel values to generate the tone scale 

function. 

21. The method claimed in claim 14, further comprising the steps 

of: 

d) forming a paxelized digital image from the pixel values; and 

e) using the paxelized digital image to generate the tone scale 

function. 

22. The method claimed in claim 17, further comprising the step of 
applying a spatial filter to the full resolution digital image. 

23. The method claimed in claim 14, further comprising the steps 

of: 

d) generating a low resolution digital image from the sparsely 
sampled high resolution digital image; 

e) applying the tone scale function to the low resolution digital 
image to form a tone scale modified low resolution digital image; 
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f) displaying the tone scale modified low resolution digital image; 

g) specifying a desired tone scale adjustment of modified low 
resolution digital image; and 

h) employing the specified tone scale adjustment to generate an 
adjusted tone scale function. 

24. The method claimed in claim 14, wherein the method is 
performed in a digital camera. 

25. The method claimed in claim 14, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein a tone scale function is generated for 
each color. 

26. The method claimed in claim 25, further comprising the steps 

of: 

d) generating a full resolution digital image from the sparsely 
sampled high resolution digital image; and 

e) applying the tone scale functions to the full resolution digital 
image to produce a tone scale adjusted full resolution digital image. 

27. A system for generating a tone scale function for a sparsely 
sampled extended dynamic range digital image, comprising: 

a) a sparsely sampled extended dynamic range image sensing 
device having fast photosites with a predetermined response to light exposure 
interspersed with slow photosites with a slower response to the same light 
exposure for producing a sparsely sampled high resolution digital image having 
fast pixel values produced by the fast photosites and slow pixel values produced 
by the slow photosites; and 
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c) means for generating the tone scale function using only slow 
pixel values from the sparsely sampled high resolution digital image. 

28. The system claimed in claim 27, further comprising: 

c) means for applying the tone scale function to the sparsely 
sampled high resolution digital image to produce a tone scale adjusted sparsely 
sampled high resolution digital image. 

29. The system claimed in claim 28, further comprising: 

d) means for generating a full resolution digital image from the 
tone scale adjusted sparsely sampled high resolution digital image to produce a 
tone scale adjusted full resolution digital image. 

30. The system claimed in claim 27, further comprising: 

c) means for generating a full resolution digital image from the 
sparsely sampled high resolution digital image; and 

d) means for applying the tone scale function to the full resolution 
digital image to produce a tone scale adjusted full resolution digital image. 

31. The system claimed in claim 27, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein the slow pixel values from only one 
color of photosite are used to generate the tone scale function. 

32. The system claimed in claim 31, wherein the different colors 
are red, green, and blue and wherein only green pixel values are used to generate 
the tone scale function. 
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33. The system claimed in claim 27, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and further comprising: 

c) means for constructing slow luminance pixel values from the 
different colored slow pixel values; and 

d) means for using the slow luminance pixel values to generate the 
tone scale function. 



34. The system claimed in claim 27, further comprising: 

c) means for forming a paxelized digital image from the slow pixel 

values; and 

d) means for using the paxelized digital image to generate the tone 

scale function. 



35. The system claimed in claim 30, further comprising means for 
applying a spatial filter to the full resolution digital image. 

36. The system claimed in claim 27, further comprising: 

c) means for generating a low resolution digital image from the 
sparsely sampled high resolution digital image; 

d) means for applying the tone scale function to the low resolution 
digital image to form a tone scale modified low resolution digital image; 

e) means for displaying the tone scale modified low resolution 

digital image; 

f) means for specifying a desired tone scale adjustment of modified 
low resolution digital image; and 

g) means for employing the specified tone scale adjustment to 
generate an adjusted tone scale function. 
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37. The system claimed in claim 27, wherein the system is in a 

digital camera. 

38. The system claimed in claim 27, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein a tone scale function is generated for 
each color. 

39. The system claimed in claim 38, further comprising: 

c) means for generating a full resolution digital image from the 
sparsely sampled high resolution digital image; and 

d) means for applying the tone scale functions to the full resolution 
digital image to produce a tone scale adjusted full resolution digital image. 

40. A system for generating a tone scale function for a sparsely 
sampled extended dynamic range digital image, comprising: 

a) a sparsely sampled extended dynamic range image sensing 
device having fast photosites with a predetermined response to light exposure 
interspersed with slow photosites with a slower response to the same light 
exposure for producing a sparsely sampled high resolution digital image having 
fast pixel values produced by the fast photosites and slow pixel values produced 
by the slow photosites; 

b) means for analyzing the fast pixel values for saturation; and 

c) means for generating the tone scale function using only slow 
pixel values from the sparsely sampled high resolution digital image if saturation 
is present in the fast pixel values, otherwise, using only fast pixel values. 

41 . The system claimed in claim 40, further comprising: 
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d) means for applying the tone scale function to the sparsely 
sampled high resolution digital image to produce a tone scale adjusted sparsely 
sampled high resolution digital image. 

42. The system claimed in claim 41, further comprising: 

e) means for generating a full resolution digital image from the 
tone scale adjusted sparsely sampled high resolution digital image to produce a 
tone scale adjusted full resolution digital image. 

43. The system claimed in claim 40, further comprising: 

d) means for generating a full resolution digital image from the 
sparsely sampled high resolution digital image; and 

e) means for applying the tone scale function to the full resolution 
digital image to produce a tone scale adjusted full resolution digital image. 

44. The system claimed in claim 40, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and wherein the pixel values from only one color of 
photosite are used to generate the tone scale function. 

45. The system claimed in claim 44, wherein the different colors 
are red, green, and blue and wherein only green pixel values are used to generate 
the tone scale function. 

46. The system claimed in claim 40, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
producing color pixel values and further comprising: 

d) means for constructing luminance pixel values from the different 
colored pixel values; and 
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e) means for using the luminance pixel values to generate the tone 

scale function. 

47. The system claimed in claim 40, further comprising: 

d) means for forming a paxelized digital image from the pixel 

values; and 

e) means for using the paxelized digital image to generate the tone 

scale function. 

48. The system claimed in claim 43, further comprising means for 
applying a spatial filter to the full resolution digital image. 

49. The system claimed in claim 40, further comprising : 

d) means for generating a low resolution digital image from the 
sparsely sampled high resolution digital image; 

e) means for applying the tone scale function to the low resolution 
digital image to form a tone scale modified low resolution digital image; 

f) means for displaying the tone scale modified low resolution 

digital image; 

g) means for specifying a desired tone scale adjustment of modified 
low resolution digital image; and 

h) means for employing the specified tone scale adjustment to 
generate an adjusted tone scale function. 

50. The system claimed in claim 40, wherein the system is in a 

digital camera. 

51. The system claimed in claim 40, wherein the image sensing 
device is a color image sensing device having differently colored photosites for 
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producing color pixel values and wherein a tone scale function is generated for 
each color. 

52. The system claimed in claim 51, further comprising: 

d) means for generating a full resolution digital image from the 
sparsely sampled high resolution digital image; and 

e) means for applying the tone scale functions to the full resolution 
digital image to produce a tone scale adjusted full resolution digital image. 

53. A computer program product for performing the method of 

claim 1. 

54. A computer program product for performing the method of 

claim 2. 

55. A computer program product for performing the method of 

claim 3. 

56. A computer program product for performing the method of 

claim 4. 

57. A computer program product for performing the method of 

claim 5. 

58. A computer program product for performing the method of 

claim 6. 

59. A computer program product for performing the method of 

claim 7. 
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. A computer program product for performing the method of 
. A computer program product for performing the method of 
. A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 
A computer program product for performing the method of 



70. A computer program product for performing the method of 



71. A computer program product for performing the method of 



72. A computer program product for performing the method of 



73. A computer program product for performing the method of 



74. A computer program product for performing the method of 



75. A computer program product for performing the method of 



76. A computer program product for performing the method of 



77. A computer program product for performing the method of 



78. A computer program product for performing the method of 



claim 26. 
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ABSTRACT 

A method of generating a tone scale function for a sparsely 
sampled extended dynamic range digital image, includes the steps of: providing a 
sparsely sampled extended dynamic range image sensing device having fast 
5 photosites with a predetermined response to light exposure interspersed with slow 
photosites with a slower response to the same light exposure; using the image 
sensing device to produce a sparsely sampled high resolution digital image having 
fast pixel values produced by the fast photosites and slow pixel values produced 
by the slow photosites; and generating the tone scale function using only slow 
1 0 pixel values from the sparsely sampled high resolution digital image. 
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I believe I am the original, first and sole inventor (if only one name is listed below) or an original, first and joint inventor (if plural names are listed 
below) of the subject matter which is claimed and for which a patent is sought on the invention entitled: 

METHOD AND APPARATUS FOR PERFORMING TONE SCALE MODIFICATIONS ON A 
SPARSELY SAMPLED EXTENDED DYNAMIC RANGE DIGITAL IMAGE 
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